1. Field of the Invention
The present invention relates to computer systems and methods in which data resources are shared among concurrent data consumers while preserving data integrity and consistency relative to each consumer. More particularly, the invention concerns an implementation of a mutual exclusion mechanism known as “read-copy update” in a user-level computing environment.
2. Description of the Prior Art
By way of background, read-copy update is a mutual exclusion technique that permits shared data to be accessed for reading without the use of locks, writes to shared memory, memory barriers, atomic instructions, or other computationally expensive synchronization mechanisms, while still permitting the data to be updated (modify, delete, insert, etc.) concurrently. The technique is well suited to multiprocessor computing environments in which the number of read operations (readers) accessing a shared data set is large in comparison to the number of update operations (updaters), and wherein the overhead cost of employing other mutual exclusion techniques (such as locks) for each read operation would be high. By way of example, a network routing table that is updated at most once every few minutes but searched many thousands of times per second is a case where read-side lock acquisition would be quite burdensome.
The read-copy update technique implements data updates in two phases. In the first (initial update) phase, the actual data update is carried out in a manner that temporarily preserves two views of the data being updated. One view is the old (pre-update) data state that is maintained for the benefit of operations that may be currently referencing the data. The other view is the new (post-update) data state that is available for the benefit of operations that access the data following the update. In the second (deferred update) phase, the old data state is removed following a “grace period” that is long enough to ensure that all executing operations will no longer maintain references to the pre-update data.
FIGS. 1A-1D illustrate the use of read-copy update to modify a data element B in a group of data elements A, B and C. The data elements A, B, and C are arranged in a singly-linked list that is traversed in acyclic fashion, with each element containing a pointer to a next element in the list (or a NULL pointer for the last element) in addition to storing some item of data. A global pointer (not shown) is assumed to point to data element A, the first member of the list. Persons skilled in the art will appreciate that the data elements A, B and C can be implemented using any of a variety of conventional programming constructs, including but not limited to, data structures defined by C-language “struct” variables.
It is assumed that the data element list of FIGS. 1A-1D is traversed (without locking) by multiple concurrent readers and occasionally updated by updaters that delete, insert or modify data elements in the list. In FIG. 1A, the data element B is being referenced by a reader r1, as shown by the vertical arrow below the data element. In FIG. 1B, an updater u1 wishes to update the linked list by modifying data element B. Instead of simply updating this data element without regard to the fact that r1 is referencing it (which might crash r1), u1 preserves B while generating an updated version thereof (shown in FIG. 1C as data element B′) and inserting it into the linked list. This is done by u1 acquiring an appropriate lock, allocating new memory for B′, copying the contents of B to B′, modifying B′ as needed, updating the pointer from A to B so that it points to B′, and releasing the lock. All subsequent (post update) readers that traverse the linked list, such as the reader r2, will see the effect of the update operation by encountering B′. On the other hand, the old reader r1 will be unaffected because the original version of B and its pointer to C are retained. Although r1 will now be reading stale data, there are many cases where this can be tolerated, such as when data elements track the state of components external to the computer system (e.g., network connectivity) and must tolerate old data because of communication delays.
At some subsequent time following the update, r1 will have continued its traversal of the linked list and moved its reference off of B. In addition, there will be a time at which no other reader process is entitled to access B. It is at this point, representing expiration of the grace period referred to above, that u1 can free B, as shown in FIG. 1D.
FIGS. 2A-2C illustrate the use of read-copy update to delete a data element B in a singly-linked list of data elements A, B and C. As shown in FIG. 2A, a reader r1 is assumed to be currently referencing B and an updater u1 wishes to delete B. As shown in FIG. 2B, the updater u1 updates the pointer from A to B so that A now points to C. In this way, r1 is not disturbed but a subsequent reader r2 sees the effect of the deletion. As shown in FIG. 2C, r1 will subsequently move its reference off of B, allowing B to be freed following expiration of the grace period.
In the context of the read-copy update mechanism, a grace period represents the point at which all running processes having access to a data element guarded by read-copy update have passed through a “quiescent state” in which they can no longer maintain references to the data element, assert locks thereon, or make any assumptions about data element state. By convention, for operating system kernel code paths, a context (process) switch, an idle loop, and user mode execution all represent quiescent states for any given CPU (as can other operations that will not be listed here).
In FIG. 3, four processes 0, 1, 2, and 3 running on four separate CPUs are shown to pass periodically through quiescent states (represented by the double vertical bars). The grace period (shown by the dotted vertical lines) encompasses the time frame in which all four processes have passed through one quiescent state. If the four processes 0, 1, 2, and 3 were reader processes traversing the linked lists of FIGS. 1A-1D or FIGS. 2A-2C, none of these processes having reference to the old data element B prior to the grace period could maintain a reference thereto following the grace period. All post grace period searches conducted by these processes would bypass B by following the links inserted by the updater.
There are various methods that may be used to implement a deferred RCU second-phase data update following a grace period, including but not limited to the use of callback processing as described in commonly assigned U.S. Pat. No. 5,727,209, entitled “Apparatus And Method For Achieving Reduced Overhead Mutual-Exclusion And Maintaining Coherency In A Multiprocessor System Utilizing Execution History And Thread Monitoring.” Another commonly used technique is to have updaters block (wait) until a grace period has completed. It should also be understood that RCU second-phase update processing may comprise actions other than freeing stale data elements. For example, a data element being updated using RCU may correspond to an operational state. In that case, the RCU second-phase data update action may comprise changing an operational state following a grace period.
Read-copy update has been used in production for many years in various operating system kernel environments, including the Linux® kernel. Multithreaded user-level (a.k.a. user-mode) application programming provides an opportunity to use read-copy update for user-level environments. Although there have been user-level implementations of RCU, these implementations place constraints on the design of the user-level application that are often intolerable in practice. Examples of such constraints include (1) requiring well-defined or special quiescent states, (2) disabling of preemption and/or interrupts, and (3) periodic interrupt, signal and/or exception processing.
An example of constraint (1) is found in a prior art user-level RCU implementation that relies on the grace period detection machinery being called periodically from a quiescent state. Without an explicitly defined quiescent state, a library function using RCU might have no idea where a quiescent state might be for the various applications running in the system. The RCU library function would need to operate correctly regardless of the design of the application calling function.
Constraint (2) is needed when an RCU reader is in the process of registering for RCU critical section processing (e.g., by incrementing or decrementing a counter that is checked during grace period detection). Otherwise, the RCU reader could be disrupted at this point due to being preempted or interrupted by preempting code that then also attempts to register for RCU critical section in competition with the RCU reader (e.g., by incrementing or decrementing the same counter). In user mode, such preempting code would be a signal handler for the RCU reader application, and disabling signal handling is very expensive on many systems. Moreover, persons skilled in the art will also recognize that upcalls, notifications, and asynchronous system traps (ASTs) (e.g., in VMS/VAX systems) may also be handled similarly to signals.
Constraint (3) includes interrupt and signal processing used to coordinate RCU reader operations with grace period detection operations by executing grace period detection code on behalf of each application that has an RCU reader. This is analogous to the grace period detection state machine used in RCU implementations for preemptible operating system kernels, where grace period detection processing is periodically executed on each CPU to coordinate with RCU readers. Constraint (3) also includes exception processing of the type described in commonly owned U.S. Patent Application Publication No. 2006/0130061. This publication discloses an RCU implementation that supports user-level operation by running an exception handler to determine if reader roll-back (and critical section reprocessing) is required for a preempted thread having an RCU critical section following a return from preemption. This recovery mechanism allows thread preemption to be considered a quiescent state, but also requires non-standard kernel modifications on most modern operating systems.
Accordingly, there is a need for a user-level implementation of read-copy update that does not place such onerous restrictions on application design.